Papers with Sparse Mixture of Experts
Turn Waste into Worth: Rectifying Top-k Router of MoE (2024.emnlp-main)
Copied to clipboard
Zhiyuan Zeng, Qipeng Guo, Zhaoye Fei, Zhangyue Yin, Yunhua Zhou, Linyang Li, Tianxiang Sun, Hang Yan, Dahua Lin, Xipeng Qiu
| Challenge: | Top-k router suffers from redundancy computation and memory costs due to unbalanced routing . some experts are overflow, where exceeding tokens are dropped, while others are empty, which are padded with zeros, negatively impacting model performance. |
| Approach: | They propose a top-k router that is unbalanced and uses a multi-gPU system to handle dropped tokens and padding. |
| Outcome: | The proposed model surpasses the top-1 router by 4.7% in terms of performance . the top-k router suffers from redundancy computation and memory costs . |
GMoE: Global Mixture of Experts with Logit Propagation (2026.acl-long)
Copied to clipboard
| Challenge: | Sparse Mixture of Experts architectures retain large memory footprints and exhibit significant redundancy, both within and across layers. |
| Approach: | They propose a sparse mixture of experts architecture that uses global experts shared across all layers and adds a Local Expert per layer for layer-specific adaptation. |
| Outcome: | The proposed architecture reduces computational cost by activating only a subset of experts per token while maintaining strong performance. |